File generation apparatus, image generation apparatus, file generation method, and storage medium

ABSTRACT

The present invention is a file generation apparatus that generates a file for generating a virtual viewpoint image. The image file generation apparatus in one aspect of the present invention has a material information accumulation unit 520 configured to acquire and accumulate material information used for generation of a virtual viewpoint image, a virtual advertisement information acquisition unit 530 configured to acquire advertisement information that is displayed on a virtual viewpoint image, and an image file generation unit 540 configured to generate an image file including material information and advertisement information.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a technique for generating a virtualviewpoint image to which additional information is assigned.

Description of the Related Art

In recent years, as a technique to reproduce an image from a camera(hereinafter, virtual camera) set within a three-dimensional virtualspace by installing a plurality of cameras at different positions andusing images captured by the plurality of cameras (hereinafter, actualcamera), there is a virtual viewpoint image generation technique(Japanese Patent Laid-Open No. 2015-45920).

However, for the virtual viewpoint image generation apparatus ofJapanese Patent Laid-Open No. 2015-45920, displaying virtual additionalinformation (for example, virtual advertisement, information relating toa player captured in a virtual viewpoint image, or the like) in avirtual viewpoint image at the time of providing the virtual viewpointimage to a viewer is not discussed. In a case where displaying a virtualadvertisement as additional information is supposed, for the virtualviewpoint image generation apparatus of Japanese Patent Laid-Open No.2015-45920, displaying a virtual advertisement is not discussed. Becauseof this, at the time of providing (displaying) a virtual viewpointimage, an advertiser will lose a chance of advertising. Further, in acase where a specific object, such as a player, is brought into focus ina captured image, there is a possibility that the area in whichadditional information (for example, advertisement sign, cheering flag,or the like) that is installed in a real space is not well focused, andtherefore, an unclear image will result. In a case where anadvertisement (hereinafter, also referred to as actual advertisement)that is installed in a real space is used, on a condition that the areacorresponding to the actual advertisement on the virtual viewpoint imageis unclear, there is a possibility that a viewer of the virtualviewpoint image cannot recognize the contents of the actualadvertisement. In this case also, an advertiser will lose a chance ofadvertising.

The present invention has been made in view of the conventional problemdescribed previously and an object thereof is to make it possible todisplay a virtual viewpoint image to which additional information isassigned at the time of displaying the virtual viewpoint image.

SUMMARY OF THE INVENTION

The file generation apparatus of the present invention has: a firstacquisition unit configured to acquire material information forgenerating a virtual viewpoint image based on a plurality of capturedimages obtained from a plurality of image capturing apparatuses; asecond acquisition unit configured to acquire advertisement informationrelating to an advertisement that is displayed on the virtual viewpointimage; and a file generation unit configured to generate a fileincluding material information acquired by the first acquisition unitand advertisement information acquired by the second acquisition unit.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing a configuration of an image generationsystem;

FIG. 2 is a diagram showing a configuration of an advertisementdatabase;

FIG. 3 is a diagram showing a GUI on which to set virtual advertisementmodel information;

FIG. 4 is a diagram showing an advertisement paste area on a virtualviewpoint image;

FIG. 5 is a diagram showing a configuration of an image file generationapparatus;

FIG. 6A and FIG. 6B are each a diagram showing a configuration of a boxin the ISO BMFF standard;

FIG. 7 is a diagram showing a configuration of an image generationapparatus;

FIG. 8 is a flowchart showing a procedure of processing to assignvarious kinds of information to an image file;

FIG. 9 is a diagram showing a format of camera parameters in an imagefile;

FIG. 10 is a diagram showing a format of information relating to aplurality of captured image in an image file;

FIG. 11 is a diagram showing a format of background model information inan image file;

FIG. 12 is a diagram showing a format of virtual advertisementinformation in an image file;

FIG. 13 is a flowchart showing a procedure of processing to generate avirtual viewpoint image to which virtual advertisement information isassigned;

FIG. 14 is a diagram showing a configuration of an image file generationapparatus;

FIG. 15 is a flowchart showing a procedure of processing to assignactual advertisement information to an image file;

FIG. 16 is a diagram showing a format of actual advertisementinformation in an image file; and

FIG. 17 is a hardware configuration diagram.

DESCRIPTION OF THE EMBODIMENTS

In the following, embodiments of the present invention are explainedwith reference to the drawings. The following embodiments are notintended to limit the present invention and all combinations of featuresexplained in the present embodiments are not necessarily indispensableto the solution of the present invention. In addition, a variety ofaspects in the scope not deviating from the gist of the presentinvention are also included in the present invention and it is alsopossible to appropriately combine part of the following embodiments. Inaddition, the virtual viewpoint image generated in each embodiment maybe a moving image or a still image.

First Embodiment

In the present embodiment, a method of assigning material informationfor generating a virtual viewpoint image and virtual advertisementinformation as additional information to an image file and a method ofgenerating and displaying a virtual viewpoint image to which a virtualadvertisement is assigned by using the file are explained. In thefollowing embodiment, an example is explained in which informationrelating to an advertisement is assigned to an image file as additionalinformation, but the additional information that is assigned is notlimited to information relating to an advertisement. The additionalinformation may be, for example, information relating to a playercaptured in a virtual viewpoint image, or information relating to agame.

<System Configuration>

FIG. 1 is a diagram showing a configuration of an image generationsystem according to the present embodiment. As shown in FIG. 1, theimage generation system according to the present embodiment comprises animage capturing apparatus 1, an advertisement database 2, an image filegeneration apparatus 3, an image generation apparatus 4, and a displaydevice 5.

The image capturing apparatus 1 includes a plurality of actual camerasand these actual cameras respectively capture an object from differentdirections. Each of the actual cameras has a camera number foridentifying the individual and transmits camera parameters and capturedimages to the image file generation apparatus 3.

The camera parameters include external parameters and internalparameters and the external parameters are supposed to be information onthe position and orientation of the actual camera and the internalparameters are supposed to be information on the focal length of theactual camera and the center coordinates of an image sensor. Further, inthe present embodiment, the lens distortion is not taken intoconsideration, but it may also be possible to use a parameter that takesthe lens distortion into consideration as needed.

The advertisement database 2 receives virtual advertisement informationcreated by an advertiser, an advertisement creator, or the like, notshown schematically, and saves the virtual advertisement information.Further, the advertisement database 2 transmits virtual advertisementinformation in response to a request from the image file generationapparatus 3. Details of the virtual advertisement information will bedescribed later.

The image file generation apparatus 3 receives captured images and thecamera parameters of all the actual cameras from the image capturingapparatus 1 and saves them. Further, the image file generation apparatus3 acquires virtual advertisement information from the advertisementdatabase 2. The image file generation apparatus 3 further generates animage file to which material information used for generation of avirtual viewpoint image and virtual advertisement information areassigned and distributes the image file to the image generationapparatus 4.

The image generation apparatus 4 generates a virtual viewpoint image towhich a virtual advertisement is assigned based on the materialinformation and the virtual advertisement information assigned to theimage file and transmits the virtual viewpoint image to the displaydevice 5. The display device 5 receives the virtual viewpoint imagetransmitted from the image generation apparatus 4 and displays thevirtual viewpoint image on a display or the like, not shownschematically.

Next, by using FIG. 2, the configuration of the advertisement database 2is explained. As shown in FIG. 2, the advertisement database 2 comprisesan advertisement information reception unit 200, an advertisementinformation accumulation unit 210, and an advertisement informationtransmission unit 220.

The advertisement information reception unit 200 receives virtualadvertisement information from an advertiser or an advertisement creatorrequested by an advertiser to create an advertisement, not shownschematically, and transmits the virtual advertisement information tothe advertisement information accumulation unit 210. The virtualadvertisement information includes information on the advertisement ID,advertiser, advertisement creator, sports name, sports stadium name,game day, display time, display start time, display end time, expirationdate, and the like. Further, the virtual advertisement informationincludes information on the advertisement model data, advertisementmaterial data, display effect, virtual viewpoint dependence,advertisement paste area on virtual viewpoint image, priority, and thelike.

The advertisement ID is an ID for identifying virtual advertisementinformation. The advertiser is a name identifying the owner of thevirtual advertisement information. The advertiser is, for example, ateam in a sports or a sports competition for which a virtual viewpointimage is generated, or a sponsor company of a program in which anvirtual viewpoint image is broadcast, and the like. The advertisementcreator is a name identifying a company or an individual who createsvirtual advertisement information by being requested by the advertiserto create an advertisement. The sports name is a name identifying asports, for example, such as soccer and baseball. The sports stadiumname is a name for identifying the sports stadium. The game day is a daythe game, such as soccer and baseball, takes place.

The display time is a time during which a virtual advertisement isdisplayed on the screen. The display start time is a time the display ofa virtual advertisement starts, and the display end time is a time thedisplay of the virtual advertisement ends. That is, the virtualadvertisement is displayed during the display time from the displaystart time until the display end time. The expiration date isinformation relating to the final day the virtual advertisement isdisplayed. After this final day passes, the information relating to thevirtual advertisement is not transmitted to the image file generationapparatus 3. The expiration date is, for example, specified by theadvertiser or determined in accordance with the contact fee of theadvertiser, or the like. Alternatively, the expiration date isdetermined by the setting by a user who views the virtual viewpointimage, the charging amount, or the like.

The advertisement model data and the advertisement material data aredata relating to the shape information and the color information on thevirtual advertisement that is actually displayed on the virtualviewpoint image. In the following, each piece of data is explainedsupplementally.

The advertisement model data is three-dimensional shape information onthe virtual advertisement and created by a general CG modeling tool. Therepresentation format is the same as a three-dimensional model used incomputer graphics and the data format is a format that has generallyprevailed, for example, the obj format. The advertisement model data maybe data described in this format or may be path information on the filedescribed in this format.

The advertisement material data is information indicating color data andtexture data of the advertisement model and created by a general CGmodeling tool like the advertisement model data. The data format is, forexample, the mtl format. The advertisement material data may be datadescribed in this format or may be path information on the filedescribed in this format.

In addition, it is also possible to register the advertisement modeldata and the advertisement texture data to the advertisement database 2by operating, for example, a GUI as shown in FIG. 3 in place of the CGmodeling tool.

In FIG. 3, symbol 300 simulates a display screen that is displayed on adisplay, not shown schematically, and symbol 310 simulates a soccerground. Further, symbol 320 simulates a plate-shaped virtual signinstalled along the goal line, symbol 321 simulates a plate-shapedvirtual sign installed along the side line (tough line), and symbol 330simulates a cylindrical virtual sign installed in the vicinity of thecorner post.

It is possible for an advertiser or an advertisement creator to registeradvertisement model data in a geometric shape by selecting the virtualsigns 320, 321, and 330 by an input device, not shown schematically.Further, it may also be possible to register an arbitrary advertisementmodel created by the above-described CG modeling tool so that theadvertisement model is displayed at an arbitrary three-dimensionalposition around the ground 310. In addition, for example, it may also bepossible to register advertisement texture data by inputting informationindicating the storage location of the file of the advertisement imagein a text box indicated by symbol 340.

The display effect is information relating to the display effect, suchas an animation and blinking, in the virtual advertisement display. Thevirtual viewpoint dependence is information identifying a virtualviewpoint independent display in which the virtual advertisement is keptbeing displayed on the virtual viewpoint image even in a case where thevirtual viewpoint is changed and a virtual viewpoint dependent displayin which the virtual advertisement is no longer displayed in a casewhere the virtual viewpoint is changed and the advertisement model goesout of the viewing angle of the virtual camera.

In a case where the virtual viewpoint independent display is selected,the advertiser or the advertisement creator selects a position at whichthe virtual advertisement is displayed on the virtual viewpoint image byusing, for example, a schematic diagram as shown in FIG. 4, or the like.In FIG. 4, symbol 400 indicates a screen on which the virtual viewpointimage is displayed, symbol 410 indicates an area in which the virtualviewpoint image is displayed, and symbol 420 and symbol 421 indicateadvertisement paste areas on the virtual viewpoint image.

It is possible for the advertiser or the advertisement creator to selectto display the virtual advertisement at the bottom of the virtualviewpoint image or to display the virtual advertisement at the top-rightby selecting the advertisement paste area 420 or 421 on the virtualviewpoint image. The advertisement paste area on the virtual viewpointimage may be an area other than the areas indicated by symbol 420 andsymbol 421. The priority is used for determining which advertisement isdisplayed with priority in a case where the displays of differentadvertisements overlap. The priority is determined, for example, inaccordance with the advertisement fee.

Returning to FIG. 2, in a case of receiving virtual advertisementinformation from the advertisement information reception unit 200, theadvertisement information accumulation unit 210 assigns theadvertisement ID to the received virtual advertisement information andsaves (accumulates) the virtual advertisement information in thedatabase. It may be possible to construct the database by analready-existing method, such as SQL. Further, in this case, theabove-described virtual advertisement information is saved in a table,but any definition may be accepted as long as it is possible to performprocessing (operation), such as a search.

The advertisement information transmission unit 220 acquires virtualadvertisement information from the advertisement informationaccumulation unit 210 in response to a request from the image filegeneration apparatus 3 and transmits the virtual advertisementinformation to the image file generation apparatus 3. In this case, theadvertisement information accumulation unit 210 searches for virtualadvertisement information from the database by using the above-describedadvertisement information (for example, advertisement ID or the like) asa key and further, transmits the searched virtual advertisementinformation to the advertisement information transmission unit 220.

Next, the configuration of the image file generation apparatus 3 isexplained by using FIG. 5. As shown in FIG. 5, the image file generationapparatus 3 comprises an image capturing information acquisition unit510, a material information accumulation unit 520, a virtualadvertisement information acquisition unit 530, an image file generationunit 540, and an image file distribution unit 550.

The image capturing information acquisition unit 510 acquires aplurality of captured images (a plurality of camera images) and cameraparameters of all the actual cameras as image capturing information. Allthe actual cameras are synchronized by common (identical) timeinformation and to the captured image, image capturing time informationis assigned. It may also be possible to transmit camera parameters alongwith images or transmit camera parameters before image capturing orseparately from images as needed.

The material information accumulation unit 520 comprises a materialinformation acquisition unit and in a case of acquiring (receiving) aplurality of captured images and camera parameters of all the actualcameras from the image capturing information acquisition unit 510, thematerial information accumulation unit 520 saves (accumulates) them. Thematerial information accumulation unit 520 further saves athree-dimensional model, such as a stadium, used at the time ofgenerating a background image of a virtual viewpoint image as backgroundmodel information. The format of the background model information mayalso be a format that has generally prevailed like the advertisementmodel and the advertisement texture of the virtual advertisementinformation and for example, the shape information is the obj format andthe color information and the texture information are the mtl format.

The virtual advertisement information acquisition unit 530 acquiresvirtual advertisement information that is assigned to an image file fromthe advertisement database 2 and transmits the acquired virtualadvertisement information to the image file generation unit 540. Theimage file generation unit 540 generates an image file for generating avirtual viewpoint image to which virtual advertisement information isassigned and transmits the generated image file to the image filedistribution unit 550. To the image file, the camera parameters of allthe actual cameras, a plurality of captured image at the time specifiedby a user, the background model information, and the virtualadvertisement information are assigned. The image file distribution unit550 distributes the image file to the image generation apparatus 4.

Here, a method of assigning the information in a case where the standardof the image file is the ISO/IEC 14496-12 (MPEG-4 Part 12) ISO basemedia format (hereinafter, ISO BMFF) standard is explained. In a casewhere described information has the same meaning, the format may beanother description method.

In the following, first, the ISO BMFF standard is explained. In the ISOBMFF standard, an image file is configured by basic units called box. Asshown in FIG. 6A, in each box, information relating to size, type, anddata is stored.

As for the box, as shown in FIG. 6B, it is also possible to storeinformation in a nested structure. As the representative kinds of box,there are ftyp (File Type Compatibility Box), moov (Movie Box), meta,and the like.

The ftyp box is arranged at the top of a file and in the ftyp box, brandinformation indicating reproduction compatibility of a file is stored.In the moov box, the frame size of data, address information indicatingthe storage position, and head information, such as the image capturingtime, are stored. As a supplement, the size of the moov box increases inproportion to the length of contents and the wait time untilreproduction lengthens, and therefore, in the ISO BMFF standard, a moofbox in the format in which header information is added for each unitobtained by dividing contents into arbitrary sections is also prepared.In the meta box, meta information is stored. In a case where the imagefile storing the information is analyzed, the moov box or the moof boxare referred to and then, the storage address of the frame correspondingto a desired time is specified and further, based on the specifiedaddress, frame data is acquired.

Next, the configuration of the image generation apparatus 4 is explainedby using FIG. 7. In a case of receiving an image file generated by theimage file generation apparatus 3, the image generation apparatus 4generates a virtual viewpoint image and further, transmits the generatedvirtual viewpoint image to the display device 5. As shown in FIG. 7, theimage generation apparatus 4 comprises an image file acquisition unit700, a material information reading unit 710, an advertisementinformation reading unit 720, and a virtual viewpoint image generationunit 730.

The image file acquisition unit 700 acquires an image file generated bythe image file generation apparatus 3 and transmits the image file tothe material information reading unit 710 and the advertisementinformation reading unit 720. The material information reading unit 710analyzes the image file and reads camera parameters of all the actualcameras, a plurality of captured images, and background modelinformation. The advertisement information reading unit 720 readsvirtual advertisement information from the received image file andtransmits the virtual advertisement information to the virtual viewpointimage generation unit 730.

The virtual viewpoint image generation unit 730 generates a virtualviewpoint image to which the virtual advertisement information isassigned by using the camera parameters of all the actual cameras, theplurality of captured images, the background model information, and thevirtual advertisement information and transmits the virtual viewpointimage to the display device 5. The display device 5 receives the virtualviewpoint image having the virtual advertisement information, which isgenerated by the image generation apparatus 4, and outputs the virtualviewpoint image to a display, not shown schematically.

<Processing Flow>

Next, by using the flowchart in FIG. 8, a method of assigning the cameraparameters of all the actual cameras, the plurality of captured images,the background model information, and the virtual advertisementinformation to the image file and the procedure of the processingthereof are explained. It is assumed that symbol “S” in explanation ofthe flowchart represents a step. This is also true with explanation ofthe subsequent flowcharts.

(S800)

At S800, the image file generation unit 540 of the image file generationapparatus 3 receives instructions to start formation of an image fileand information on the start time and the end time of a virtualviewpoint image from a user, not shown schematically. In the following,the start time of the virtual viewpoint image is referred to as an imagegeneration start time, but this is not the time at which generation ofthe virtual viewpoint image is started but the time of image capturingof a captured image that is used in a case where the virtual viewpointimage is generated. Further, the end time of the virtual viewpoint imageis referred to an image generation end time, but this is the same as inthe case with the image generation start time.

Specifically, first, the image file generation unit 540 distributes animage of an arbitrary actual camera saved in the material informationaccumulation unit 520 to a viewing environment, not shown schematically,so that it is possible for the user to specify a time at which the userdesires to generate the virtual viewpoint image. Next, the image filegeneration unit 540 receives information relating to the imagegeneration start time, the image generation end time, and the number offrames for which an image file is generated, which are input by theuser. Upon receipt of the start time and the end time, the image filegeneration unit 540 generates an image file for generating a movingimage. In a case where an image file for generating a still image isgenerated, the image file generation unit 540 receives only the starttime and sets the number of frames to one.

Then, the image file generation unit 540 generates an image file. Inthis case, at the time of generating an image file for each frame, theimage file generation unit 540 continues to generate, upon receipt ofthe information relating to the start time, an image file for each frameuntil receiving end instructions. Alternatively, at the time ofgenerating an image file by collecting a plurality of frames, the imagefile generation unit 540 generates, upon receipt of the informationrelating to the start time and the end time, one image file in themeanwhile.

Further, upon receipt of instructions to start generation of an imagefile, the image file generation unit 540 describes, for example, acharacter string of fvpm as brand information in ftyp and explicitlyindicates that information for virtual viewpoint image generation issaved in the image file.

(S810)

At S810, the image file generation unit 540 acquires camera parametersrelating to each actual camera from the material informationaccumulation unit 520 and assigns the camera parameters to the imagefile. Here, by using FIG. 9, the format by which the camera parameter ofthe actual camera is described is explained supplementally.

The camera parameter is described in the meta box of the ISO BMFFstandard. As shown in FIG. 9, in the meta information, informationrelating to data type and number of cameras is described. In data type,a character string of camera_parameter is described as informationindicating a camera parameter. In number of cameras, the number ofactual cameras is described.

In data information, external parameters and internal parameterscorresponding to the number of actual cameras are described. Forexample, in a case where there are N actual cameras, the values of allthe elements of a matrix representing the external parameters and theinternal parameters of the first actual camera are described. Followingthis, the values of all the elements of a matrix representing theexternal parameters and the internal parameters of the second actualcamera are described. After this, in ascending order, the externalparameters and the internal parameters of the actual cameras aredescribed similarly until the Nth actual camera is reached.

(S820)

At S820, the image file generation unit 540 acquires informationrelating to a plurality of captured images from the start time until theend time for generating a virtual viewpoint image from the materialinformation accumulation unit 520 and assigns the information to theimage file. Here, by using FIG. 10, the format by which informationrelating to a plurality of captured images is described is explainedsupplementally.

In a case where an image file is generated by collecting a plurality offrames, information relating to a plurality of captured images isdescribed as the moov box or the moof box of the ISO BMFF standard. Inthis case, in moov information or moof information, information on widthof camera image, height of camera image, number of actual cameras, totalnumber of frames of camera image, image capturing start time, and imagecapturing end time is described.

In a case of the moov box, in the data information, pixel information onthe camera images of all the actual cameras is described for each frame.Further, in a case of the moof box, in the data information, informationon the camera images of all the actual cameras corresponding to oneframe is described in one file. In a case where an image file isgenerated for each frame, the image file generation unit 540 describesinformation similarly in the moof box.

(S830)

At S830, the image file generation unit 540 acquires background modelinformation used at the time of generating a background image forgenerating a virtual viewpoint image from the material informationaccumulation unit 520 and assigns the information to the image file.Here, by using FIG. 11, the format by which background model informationis described is explained supplementally.

The background model information is also described in the meta box ofthe ISO BMFF standard. In the meta information, information relating todata type is described. In data type, as information indicatingbackground model information, a character string of background_model isdescribed. In the data information, information (Obj file path)indicating the saving destination of file of the obj format of thebackground model and information (Mtl file path) indicating the savinginformation of the file of the mtl format are described. Alternatively,it may also be possible to describe the contents of these files as theyare.

(S840)

At S840, the virtual advertisement information acquisition unit 530acquires virtual advertisement information from the advertisementdatabase 2 and transmits the virtual advertisement information to theimage file generation unit 540. The image file generation unit 540assigns the received virtual advertisement information to the imagefile. The virtual advertisement information acquisition unit 530acquires the virtual advertisement information from the advertisementdatabase 2 based on sports name, sports stadium name, game day, andexpiration date.

In the following, specific explanation is given. In a case where a useractivates the image file generation apparatus 3, the user registersinformation relating to the name of the sports that is captured, thename of the sports stadium, and the game day the sports takes place tothe material information accumulation unit 520. The virtualadvertisement information acquisition unit 530 acquires virtualadvertisement information relating to the same game day as that of theinformation relating to the game day registered in the materialinformation accumulation unit 520, the same sports name as that of theinformation, and the same sports stadium name as that of the informationfrom the advertisement database 2. In this case, the virtualadvertisement information whose expiration date has passed is notacquired.

Next, by using FIG. 12, the format by which the image file generationunit 540 assigns virtual advertisement information to an image file isexplained supplementally. The virtual advertisement information is alsodescribed in the meta box of the ISO BMFF standard. In the metainformation, information relating to data type, number ofadvertisements, data size of each piece of advertisement information,and address of advertisement database is described.

In data type, as information indicating virtual advertisementinformation, a character string of virtual_advertising is described. Innumber of advertisements, the number of assigned advertisements isdescribed. In data size, the data size of the advertisement is describedfor each advertisement (corresponding to the number of advertisements).In address of advertisement database, as address information forconnecting to the advertisement database 2, for example, URL or the likeis described.

In a case where there is no information on address of advertisementdatabase, in the data information, as advertisement information,advertisement ID, advertiser, advertisement creator, sports name, sportsstadium name, game day, display time, display start time, and displayend time are described. Further, in the data information, asadvertisement information, information on expiration date, advertisementmodel data, advertisement material data, display effect, virtualviewpoint dependence, advertisement paste area on virtual viewpointimage, and priority is described. Further, in a case where there isinformation on address of advertisement database (that is, in a casewhere URL is described in address of advertisement database), in thedata information, only advertisement ID is described. In a case whereonly advertisement ID is described as above, in displaying virtualadvertisement information, the advertisement database 2 is accessed byusing the address information and the virtual advertisement informationon advertisement ID is acquired.

(S850)

At S850, the image file generation unit 540 transmits the generatedimage file to the image file distribution unit 550. Then, the image filedistribution unit 550 distributes the image file to the image generationapparatus 4. That is, the image file generation unit 540 generates animage file including the camera parameters, the information relating tothe captured image, the background model information, and the virtualadvertisement information, which are acquired at S810 to S840. The imagefile is generated for each frame based on the information relating tothe start time received at S800. Further, in a case where an image fileis generated based on the information relating to the start time and theend time received at S800, the image file is generated as an image filehaving data indicating the image of one frame, which corresponds to aplurality of collected frames.

Following the above, by using the flowchart in FIG. 13, the method ofgenerating a virtual viewpoint image to which virtual advertisementinformation is assigned by using the above-described image file and theprocedure of the processing thereof are explained.

(S1300)

At S1300, the image file acquisition unit 700 of the image generationapparatus 4 acquires an image file from the image file generationapparatus 3. Then, the image file acquisition unit 700 analyzes theimage file and reads the kind of image file, the camera parameters ofall the actual cameras, the plurality of captured images, the backgroundmodel information, and the virtual advertisement information. First, aviewer, not shown schematically, selects to generate an image file bycollecting a plurality of frames or to generate an image file for eachframe.

In a case of generating an image file by collecting a plurality offrames, the viewer sets the start time and the end time of a scene forwhich a virtual viewpoint image is generated. In order to set the starttime and the end time, the image file acquisition unit 700 acquires anarbitrary actual camera image at the time specified by the viewer fromthe image file generation apparatus 3 and displays the actual cameraimage on a display or the like, not shown schematically. The viewer setsthe start time and the end time of the scene for which a virtualviewpoint image is generated while watching the displayed image.

The image file acquisition unit 700 transmits instructions to startimage file generation by the viewer and the information relating to thestart time and the end time to the image file generation apparatus 3 andreceives the image file to which virtual advertisement information isassigned, which is generated by the image file generation apparatus 3.Then, the image file acquisition unit 700 transmits the received imagefile to the material information reading unit 710 and the advertisementinformation reading unit 720.

Further, in a case where an image file is generated for each frame, theimage file acquisition unit 700 transmits instructions to start imagefile generation by the viewer and the information relating to the starttime and receives the image file to which virtual advertisementinformation is assigned, which is generated by the image file generationapparatus 3. The image file acquisition unit 700 keeps receiving thegenerated image file until transmitting end instructions and transmitsthe received image file to the material information reading unit 710 andthe advertisement information reading unit 720.

(S1310)

At S1310, the material information reading unit 710 scans the image fileand checks that the character string described in the ftyp box is fvpm.In a case where the character string described in the ftyp box is notfvpm, the material information reading unit 710 aborts reading of thefile and in a case where the character string described in the ftyp boxis fvpm, the material information reading unit 710 continues to read thefile.

The material information reading unit 710 scans the image file and in acase where the kind of box is the meta box and the character stringdescribed in data type of the meta information is camera_parameter, thematerial information reading unit 710 reads the camera parameters of theactual camera. In this case, the material information reading unit 710reads the number of actual cameras from the meta information and readsthe external parameters and the internal parameters corresponding to thenumber of actual cameras from the data information.

Further, the material information reading unit 710 scans the image fileand in a case where the kind of box is the moov box or the moof box, thematerial information reading unit 710 reads the information relating tothe plurality of captured images. The information relating to the widthand height of the image, the number of cameras, and the number offrames, which is used for reading multi-camera image information, isacquired from the moov box or the moof box. The material informationreading unit 710 cuts out the number of images corresponding to thenumber of cameras for each frame from the data information, whichcorresponds to the number of pixels (that is, image width×height), andreads the multi-camera image information.

In a case of completing reading of the multi-camera image information,the material information reading unit 710 generates an object model usedfor generation of a virtual viewpoint image based on the multi-cameraimage information. The object model includes shape information and colorinformation. Further, the processing to generate these pieces ofinformation mainly includes processing to estimate a three-dimensionalshape of the object model and processing to estimate the color of thethree-dimensional shape. In the following, each piece of processing isexplained.

In order to estimate the three-dimensional shape of the object model,first, the material information reading unit 710 estimates the objectarea from each actual camera image. This object area is estimated by,for example, a general method, such as a background difference method inwhich a difference from the background image captured in advance iscalculated from the captured image obtained by capturing the objectmodel and the area whose difference is more than or equal to apredetermined threshold value is estimated as the object area.

Then, the material information reading unit 710 estimates thethree-dimensional shape of the object model by using the object areasand the camera parameters of all the actual cameras. For estimation ofthe three-dimensional shape, for example, the Visual Hull method or thelike is used. In the present embodiment, this shape of the object modelis represented by a three-dimensional point cloud (set of points havingthree-dimensional coordinates).

Following the above, the color of each point configuring thethree-dimensional point cloud is estimated. In order to estimate thecolor, first, a distance image is generated in which the distance fromthe actual camera to the object surface is taken to be the distance.Next, for each point, the distance value acquired by performingconversion into each camera coordinate system is compared with the pixelvalue (distance) of the pixel projected onto the distance image and in acase where the difference is less than or equal to a predeterminedthreshold value, it is determined that the point is viewed from theactual camera. Then, in all the cameras from which it is determined thatthe point is viewed, the color of the pixel of the projectiondestination is calculated and the calculated average of the pixel colorsis estimated as the color of the point. By the above processing, it ispossible to acquire the object model whose each point is colored.

Further, the material information reading unit 710 scans the image fileand in a case where the kind of box is the meta box and the characterstring described in data type of the meta information isbackground_model, the material information reading unit 710 reads thebackground model information. In this case, the material informationreading unit 710 acquires the background model information (shapeinformation on the background model and color information on thebackground model) by reading the paths of the obj format file and themtl format file described in the data information.

The shape information on the background model is acquired by reading theobj format file. Further, the color information on the background modelis acquired by reading the mtl format file. In a case where there is nomtl format file of the background model, it may also be possible togenerate texture that is mapped to the background model from thebackground image and take the texture as color information. In order togenerate texture, first, the normal to each surface is calculated basedon the shape information on the background model and the actual camerathat faces most squarely is selected by comparing the calculated normalto each surface and the orientation of the actual camera. Then, thethree vertexes configuring the surface are projected onto the backgroundimage of the actual camera and the image within the triangle formed bythe three vertexes is taken as the texture of each surface.

In a case where reading of material information is completed andfurther, estimation of the shape information and the color informationon the object model is completed, the material information reading unit710 transmits the external parameters and the internal parameters of theactual camera, the plurality of captured images, the object model, andthe information on the background model to the virtual viewpoint imagegeneration unit 730.

(S1320)

At S1320, the advertisement information reading unit 720 scans the imagefile and reads the virtual advertisement information, and transmits theadvertisement model data to the virtual viewpoint image generation unit730. First, the advertisement information reading unit 720 checks thatthe character string described in the ftyp box is fvpm. Theadvertisement information reading unit 720 aborts reading of the file ina case where the character string described in the ftyp box is not fvpmand continues to read the file in a case where the character stringdescribed in the ftyp box is fvpm.

The advertisement information reading unit 720 scans the image file andreads the virtual advertisement information in a case where the kind ofbox is the meta box and the character string described in data type ofthe meta information is virtual_advertising. In a case where there is noinformation on address of advertisement database in the metainformation, the advertisement information reading unit 720 readsinformation on advertisement ID, advertiser, advertisement creator,sports name, sports stadium name, game day, display time, display starttime, display end time, and expiration date described in the datainformation as the virtual advertisement information. Further, theadvertisement information reading unit 720 reads information onadvertisement model data, advertisement material data, display effect,virtual viewpoint dependence, advertisement paste area on virtualviewpoint image, and priority described in the data information as thevirtual advertisement information. Furthermore, in a case where URL ofthe advertisement database is described in address of advertisementdatabase, the advertisement information reading unit 720 accesses theadvertisement database 2 with the URL and acquires the virtualadvertisement information on advertisement ID described in the datainformation.

After reading the virtual advertisement information in this manner, theadvertisement information reading unit 720 generates the advertisementmodel data. Like the object model and the background model, theadvertisement model also includes shape information and colorinformation. The shape of the advertisement model is acquired by readingthe obj format file obtained as the advertisement model data. Further,the color information on the advertisement model is acquired by readingthe mtl format file obtained as the advertisement material data.

(S1330)

At S1330, the virtual viewpoint image generation unit 730 generates avirtual viewpoint image to which advertisement information is assignedby using information relating to the camera parameters of all the actualcameras, the plurality of captured images, the object model, thebackground model, the advertisement model, and the parameters of thevirtual camera. The virtual viewpoint image generated by the imagegeneration apparatus 4 is transmitted to the display device 5 anddisplayed on a display or the like. The parameters of the virtual cameramay be the same as those of an arbitrary actual camera and an arbitraryviewpoint position, line-of-sight vector, and viewing angle may be set.

In the following, a method of generating a virtual viewpoint image(generation method of a virtual viewpoint image) from theabove-described information is explained supplementally. Here, thegeneration method of a virtual viewpoint image is changed in accordancewith the information relating to the virtual viewpoint dependence of theadvertisement information.

In a case where the display method depends on the virtual viewpoint, byperforming rendering of the background model, the object model, and theadvertisement model by a general CG method, it is possible to generate avirtual viewpoint image. In addition, in a case where as the displayeffect of the advertisement information, for example, “blinking display,display interval n frames” or the like is set, it is possible togenerate an image that simulates the blinking effect by abortingrendering of the advertisement model for every n frames of the virtualviewpoint image. In a case of the display method that depends on thevirtual viewpoint, the virtual advertisement model outside the viewingangle of the virtual camera is not displayed on the virtual viewpointimage.

Further, in a case where the display method does not depend on thevirtual viewpoint, it is possible to generate a virtual viewpoint imageby pasting the texture of the virtual advertisement model to theadvertisement paste area on the virtual viewpoint image obtained byperforming rendering of the background model and the object model. Thedisplay of the virtual advertisement information is started from thetime after the display start time acquired as the advertisementinformation elapses relatively from the image generation start time ofthe scene for which the virtual viewpoint image is generated, which isset by a viewer at S1300, and the virtual advertisement information isdisplayed until the image generation end time is reached, that is,during the time corresponding to the display time.

In a case where the display start time is not set, the display isstarted from the image generation start time and in a case where thedisplay end time is not set, the display is continued until the imagegeneration end time is reached. In a case where the display time is notset, the display is continued from the display start time to the displayend time. Further, in a case where the generation day of the virtualviewpoint image in the image file is after the expiration date of thevirtual advertisement information passes, the virtual advertisementinformation is not displayed. In this case, on a condition that there isanother piece of virtual advertisement information within the expirationdate, only the virtual advertisement information within the expirationdate is displayed and on a condition that there is no advertisementinformation within the expiration date, a virtual viewpoint image withno virtual advertisement display is displayed. In addition, in a casewhere virtual advertisements of different advertisers overlap on onevirtual viewpoint image, by using priority acquired as virtualadvertisement information, the virtual advertisement with high priorityis displayed preferentially.

Further, in a case of generating a virtual viewpoint image at the nexttime, the image generation apparatus 4 returns the processing to S1310and generates a virtual viewpoint image at the new time and transmitsthe virtual viewpoint image to the display device 5.

As explained above, it is possible for the image file generationapparatus 3 of the present embodiment to generate an image file forviewing a virtual viewpoint image at the specified time. Further, it ispossible to assign virtual advertisement information to the image file.Furthermore, it is possible for the image generation apparatus 4 togenerate and display a virtual viewpoint image to which virtualadvertisement information is assigned by acquiring an image file towhich virtual advertisement information is assigned from the image filegeneration apparatus 3. In a case where an image file to which virtualadvertisement information is assigned is generated by the image filegeneration apparatus 3 of the present embodiment, there is an effect asfollows. For example, in a case where an advertiser or a user desires tosearch for a virtual viewpoint image to which an advertisement of aspecific company or the like is assigned, it is possible to easilysearch for the virtual viewpoint image by using the information includedin the virtual advertisement information. Further, the virtualadvertisement information and the material information are included inone file, and therefore, the processing relating to a call of data issimplified compared to a case where the virtual advertisementinformation and the material information are acquired separately.

In the present embodiment, the procedure of the processing is notnecessarily limited to this and for example, it may also be possible toperform the processing to acquire the material information and theprocessing to acquire the virtual advertisement information by reversingthe order. Further, it may also be possible to encrypt the virtualadvertisement information on the image file by an already-existingencryption method.

Second Embodiment

In the first embodiment described above, in order to generate texture ofthe background model, a plurality of captured images is used. However,in the plurality of captured images (actual camera images), players arebrought in focus but the image of the area in which a sign is arrangedis an image not so well focused.

Consequently, in the present embodiment, a method is explained in whichthe image of an actual sign area of a virtual viewpoint image isdisplayed clearly by assigning actual advertisement information to animage file as additional information and making use of the actualadvertisement information also at the time of background modelgeneration. The actual advertisement in the present embodiment refers toan existing sign or display that is installed in a sports stadium or astadium, which is a real space. In the following embodiment, an exampleis explained in which actual advertisement information is assigned to animage file as additional information that is made use of at the time ofbackground model generation, but the additional information that isassigned is not limited to actual advertisement information. Additionalinformation may be, for example, information relating to a cheering flagthat is installed in the real space, or the like.

<System Configuration>

FIG. 14 is a diagram showing a configuration of an image generationsystem according to the present embodiment. As shown in FIG. 14, theimage generation system according to the present embodiment comprisesthe image capturing apparatus 1, the advertisement database 2, the imagefile generation apparatus 3, the image generation apparatus 4, and thedisplay device 5. In the present embodiment, the configuration of theimage capturing apparatus 1 and the display device 5 is the same as theconfiguration of the above-described first embodiment.

The image file generation apparatus 3 further comprises an actual signarea setting unit 1400 other than the image capturing informationacquisition unit 510, the material information accumulation unit 520,the virtual advertisement information acquisition unit 530, the imagefile generation unit 540, and the image file distribution unit 550described above. In relation to this, to the image file generation unit540, processing to assign actual advertisement information is added. Theconfiguration of the advertisement database 2 and the image generationapparatus 4 is the same as the configuration of the above-describedfirst embodiment, but the operation of each unit is different from thatof the above-described first embodiment because actual advertisementinformation is controlled.

In the present embodiment, the actual advertisement information isinformation on advertisement ID, advertiser, advertisement creator,sports name, sports stadium name, game day, actual advertisement image,actual camera ID, actual sign area, and the like. Here, actualadvertisement image is an image, such as a company logo, and the sameimage as the design of the sign installed actually and actual sign areainformation is a label image indicating the sign area in each actualcamera image. It is assumed that the image size is the same as that ofthe actual camera image and the label value of each pixel corresponds toadvertisement ID. For example, it is assumed that the label value 0 isbackground, the label value 1 is advertisement ID1, and the label value2 is advertisement ID2. That is, it is assumed that a set of the labelvalue 1 is the sign area of advertisement ID1. Further, advertisement IDand advertiser are associated with each other in the advertisementdatabase 2.

Information other than the information on actual sign area is registeredto the advertisement database 2 by an advertiser or an advertisementcreator, which is not shown schematically, before an image file isgenerated. Further, the actual sign area information is set by theactual sign area setting unit 1400

<Processing Flow>

Next, a method of assigning actual advertisement information to an imagein accordance with the ISO BMFF standard and a procedure of theprocessing thereof are explained by using a flowchart in FIG. 15.

(S1510)

At S1510, the image capturing information acquisition unit 510 acquiresa plurality of captured images of a sign actually installed in thesports stadium and saves (accumulates) in the material informationaccumulation unit 520.

(S1520)

At S1520, the actual sign area setting unit 1400 acquires the pluralityof captured images from the material information accumulation unit 520and extracts the actual sign areas from the camera images of all theactual cameras. Then, the actual sign area setting unit 1400 transmitsactual sign area information to the image file generation unit 540.

Further, it is possible to extract the actual sign area by, for example,a template matching method. In this case, the actual sign area settingunit 1400 first acquires the actual sign information on the advertiserwho has installed the actual sign in the sports stadium from theadvertisement database 2 based on sports stadium name and game day.Next, the actual sign area setting unit 1400 extracts a similar imagearea from the camera image of the actual camera by using the actualadvertisement image including in the actual advertisement information asa template and extracts the similar image area as the actual sign areain the actual camera.

Here, extraction of the actual sign area by template matching isexplained, but it is also possible to manually extract an arbitrary areaof the image by using an input device, such as a mouse, and take thearea as the actual sign area. The actual advertisement informationincluding the actual sign area information is transmitted to the imagefile generation unit 540.

(S1530)

At S1530, the image file generation unit 540 generates an image file towhich actual advertisement information is also assigned and transmitsthe image file to the image file distribution unit 550. Here, by usingFIG. 16, the format by which actual advertisement information isdescribed is explained supplementally.

Actual advertisement information is also described as the meta box ofthe ISO BMFF standard. In the meta information, information relating todata type and advertiser table is described. In data type, asinformation indicating actual advertisement information, a characterstring of actual_advertising is described. In advertiser table, acorrespondence between the label value of actual sign area andadvertisement ID is described. In the data information, as actualadvertisement information, information on advertisement ID, advertiser,advertisement creator, sports name, sports stadium name, game day,actual advertisement image, camera ID corresponding to the number ofcameras, actual sign area, and priority.

Following the above, by using the above-described image file, a methodof generating a virtual viewpoint image that makes use of actualadvertisement information and a procedure of the processing thereof areexplained. First, like S1300 of the first embodiment, the image fileacquisition unit 700 acquires an image file from the image filegeneration apparatus 3. Then, the image file acquisition unit 700transmits the received image file to the material information readingunit 710 and the advertisement information reading unit 720.

The processing of the material information reading unit 710 is the sameas that at S1310, but in a case where there is actual advertisementinformation, the processing to generate color information on thebackground model is different. In the following, this processing isexplained supplementally, but first, the processing to read actualadvertisement information is explained. First, the material informationreading unit 710 scans the image file and checks that the characterstring described in the ftyp box is fvpm.

The material information reading unit 710 aborts reading of the file ina case where the character string described in the ftyp box is not fvpmand continues reading of the file in a case where the character stringdescribed in the ftyp box is fvpm. The material information reading unit710 scans the image file and in a case where the kind of box is the metabox and the character string described in data type of the metainformation is actual_advertising, reads actual advertisementinformation. In this case, the material information reading unit 710reads information on advertisement ID, advertiser, advertisementcreator, sports name, sports stadium name, game day, actualadvertisement image, and actual sign area of all the actual cameras,which are described in the data information.

The color information on the background model in a case where there isactual advertisement information is generated by using the actualadvertisement image. First, the material information reading unit 710calculates the normal to each surface based on the shape information onthe background model and selects the actual camera that faces mostsquarely by comparing the calculated normal to each surface and theorientation of the actual camera. Then, the three vertexes configuringthe surface are projected onto the camera image of the actual camera andin a case where the three vertexes are located within the actual signarea, the three vertexes are projected onto the advertisement image andthe image within the triangle is taken as the texture of the surface. Byperforming this processing on all the surfaces of the background model,it is possible to generate the color information on the background modelthat makes use of the advertisement image.

Like S1320, the advertisement information reading unit 720 scans theimage file and reads virtual advertisement information and transmits thevirtual advertisement information to the virtual viewpoint imagegeneration unit 730. Like S1330, the virtual viewpoint image generationunit 730 generates a virtual viewpoint image to which advertisementinformation is assigned by using the information on the cameraparameters of the actual cameras, the object model, the background modelgenerated from the actual advertisement information, and the virtualadvertisement model. The generated virtual viewpoint image istransmitted to the display device 5 and displayed on a display or thelike.

By the present embodiment, it is possible for a user, not shownschematically, to acquire an image file to which actual advertisementinformation is also assigned. Further, it is possible to make clear theactual sign image on the virtual viewpoint image by acquiring the actualadvertisement information based on the image file and generating thebackground model by using the actual advertisement image.

Third Embodiment

In the above-described second embodiment, the method of making clear theactual sign image on the virtual viewpoint image by using the actualadvertisement information is explained. However, with this method, thereis a case where a virtual advertisement image is displayed on the actualsign area.

Consequently, in the present embodiment, a method is explained thatperforms control so that a virtual advertisement image is not displayedon the actual sign area by using virtual viewpoint virtual advertisementinformation and actual advertisement image, both assigned to an imagefile. Due to this, an advertiser is no longer deprived of a chance ofadvertising by a different advertiser. Further, in a case where theadvertiser of the virtual advertisement and the advertiser of the actualadvertisement are the same advertiser, it is possible to increase thechance of advertising.

<System Configuration>

The configuration of an image generation system according to the presentembodiment is the same as that of the above-described second embodiment.However, the operation of the virtual viewpoint image generation unit730 in the image generation apparatus 4 is different from that of theabove-described second embodiment. In the following, the operation ofthe virtual viewpoint image generation unit 730 is explained.

In a case of receiving both virtual advertisement information and actualadvertisement information, the virtual viewpoint image generation unit730 operates as follows. As explained in the above-described secondembodiment, first, the virtual viewpoint image generation unit 730determines whether or not each surface of the background model isincluded in the actual sign area. Next, the virtual viewpoint imagegeneration unit 730 calculates the actual sign area on the virtualviewpoint image by projecting the surface included in the actual signarea of the background model onto the virtual viewpoint. Further, thevirtual viewpoint image generation unit 730 calculates the virtualadvertisement image on the virtual viewpoint image by projecting thevirtual advertisement model onto the virtual viewpoint.

Then, in a case where the actual sign area and the virtual advertisementarea overlap, the overlap of the actual sign and the virtual sign isavoided by shifting the virtual advertisement area to the outside of therectangle surrounding the actual sign area. Alternatively, it may alsobe possible to preferentially display advertisement whose priorityincluded in the advertisement information is high. Further, bycalculating the object area on the virtual viewpoint image by projectingthe object model onto the virtual viewpoint image and determining theoverlap of the object area and the virtual advertisement area or theactual sign area, it is also possible to perform control so that theobject area and the advertisement area do not overlap on the virtualviewpoint image.

By the present embodiment, in a case where virtual advertisementinformation and actual advertisement information are assigned to animage file, it is possible to prevent the virtual advertisement imagefrom being displayed on the actual sign area. Due to this, it ispossible to display the advertisement without depriving the advertisersof the respective chances of advertising.

It is possible to implement the apparatus, such as the image filegeneration apparatus and the image generation apparatus explained in thefirst, second, and third embodiments by the same configuration as theconfiguration of a general computer as shown in FIG. 17. A CPU (CentralProcessing Unit) 1701 controls the operation of the entire apparatus byusing input data or computer programs stored in a ROM 1702 and a RAM1703, to be described later. It may also be possible to comprise onepiece or a plurality of pieces of dedicated hardware different from theCPU 1701 or a GPU (Graphics Processing Unit) and for the dedicatedhardware or the GPU to perform at least part of the processing by theCPU 1701. As the dedicated hardware, for example, there are an ASIC(Application Specific Integrated Circuit), a DSP (Digital SignalProcessor), and the like.

The ROM (Read-Only Memory) 1702 stores setting parameters for settingeach unit of the apparatus, a boot program, and the like. The RAM(Random Access Memory) 1703 has a storage area temporarily storingcomputer programs and data read from an external storage device 1704 anddata received from the outside via a general-purpose interface 1705, tobe described later. Further, the RAM 1703 is used as a storage area usedby the CPU 1701 in a case of performing various kinds of processing.

The external storage device 1704 is a storage device storing variouskinds of data and the like necessary for the CPU 1701 to perform variouskinds of processing and is, for example, an HDD (Hard Disk Drive) or thelike. The general-purpose interface 1705 is an interface forcommunicating with an external apparatus 1706 and is, for example, a USB(Universal Serial Bus) interface or the like.

Further, in the above-described configuration, each function of theapparatuses and devices is implemented by the CPU 1701 reading andexecuting programs capable of implementing the flows (flowcharts) of theprocessing shown in FIG. 8, FIG. 13, FIG. 15 and the like.

Furthermore, in the first, second, and third embodiments, the example isexplained in which as additional information that is assigned to animage file, advertisement information (virtual advertisement informationand actual advertisement information) is assigned, but the example isnot limited to this. As another example of additional information thatis assigned to an image file, a case is explained where information(hereinafter, referred to as player information) relating to a playercaptured in virtual viewpoint information is assigned. In a case whereplayer information is assigned to an image file in accordance with theISO BMFF standard, in the meta box, information relating to data type,number of pieces of data, data size of each piece of information,address of player information database is described. Here, data type isinformation indicating that additional information is informationrelating to a player, number of pieces of data is the number of piecesof player information, and data size of each piece of information isinformation indicating the data size of the player information.Information relating to address of player information database isaddress information for connecting to a database in which playerinformation is accumulated. In a case where address information is notdescribed in the meta box, player information ID, provider of playerinformation, creator of player information, sports name, sports stadiumname, game day, display time, display start time, and display end timeare described in the data box. Further, in a case where addressinformation is not described in the meta box, information on expirationdate, player information model data, player information material data,display effect, virtual viewpoint dependence, player information pastearea on virtual viewpoint image, and priority is described in the databox. In a case where address information is described in the meta box,only player information ID is described in the data box As above, it ispossible to apply the present invention to information other thanadvertisement information.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

According to the present invention, in a case where a virtual viewpointimage is displayed, it is possible to display a virtual viewpoint imageto which additional information is assigned.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2018-237547, filed Dec. 19, 2018, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. A file generation apparatus comprising: one ormore hardware processors; and one or more memories storing one or moreprograms configured to be executed by the one or more hardwareprocessors, the one or more programs including instructions for: (1)acquiring material information for generating a virtual viewpoint imagebased on a plurality of captured images obtained from a plurality ofimage capturing apparatuses, the acquired material information includinginformation for specifying positions and orientations of the pluralityof image capturing apparatuses, wherein the positions of the pluralityof image capturing apparatuses differ from each other and theorientations of the plurality of image capturing apparatuses differ fromeach other; (2) acquiring additional information for specifying displayinformation that is included in the virtual viewpoint image generatedbased on the material information; and (3) generating a file, forgenerating the virtual viewpoint image including the displayinformation, the file including (a) the acquired material informationand (b) the acquired additional information.
 2. The file generationapparatus according to claim 1, wherein the additional informationincludes data representing a shape of the predetermined information anddata representing texture of the predetermined information.
 3. The filegeneration apparatus according to claim 1, wherein the additionalinformation includes an address indicating a location where (i) datarepresenting a shape of the predetermined information and (ii) datarepresenting texture of the predetermined information are stored.
 4. Thefile generation apparatus according to claim 1, wherein the additionalinformation includes information indicating an area in which thepredetermined information is displayed on the virtual viewpoint image.5. The file generation apparatus according to claim 1, wherein theadditional information includes information relating to an expirationdate indicating a period during which the predetermined information isdisplayed on the virtual viewpoint image.
 6. The file generationapparatus according to claim 1, wherein the material informationincludes at least one of a plurality of captured images obtained bycapturing an object by the plurality of image capturing apparatuses,parameters relating to the plurality of image capturing apparatuses, anda background model for generating a background image on a virtualviewpoint image.
 7. The file generation apparatus according to claim 1,wherein the generated file is a file in accordance with an ISO BMFF (ISObase media format) standard.
 8. The file generation apparatus accordingto claim 7, wherein the generated file is a file in which the materialinformation and the additional information are stored in an identicalbox.
 9. The file generation apparatus according to claim 1, wherein thedisplay information is an advertisement included in the virtualviewpoint image.
 10. The file generation apparatus according to claim 9,wherein the additional information includes at least one of (i)information relating to an actual advertisement that is installed in areal space and (ii) information relating to a virtual advertisement thatis added virtually to a virtual viewpoint image.
 11. The file generationapparatus according to claim 9, wherein the additional informationincludes at least one of (i) information indicating an advertiser of theadvertisement, (ii) information indicating a creator who has created theadvertisement, (iii) information indicating a display time during whichthe advertisement is displayed on the virtual viewpoint image, and (iv)information indicating priority of the advertisement.
 12. The filegeneration apparatus according to claim 1, wherein the additionalinformation includes information indicating the three-dimensionalposition where the display information is placed.
 13. The filegeneration apparatus according to claim 1, wherein the three-dimensionalposition where the display information is placed is determined bydesignating a position in an image representing an area captured by theplurality of image capturing apparatuses.
 14. The file generationapparatus according to claim 13, wherein the three-dimensional positionwhere the display information is placed is determined by selecting aposition from among predetermined positions in an image representing anarea captured by the plurality of image capturing apparatuses.
 15. Animage generation apparatus comprising: one or more hardware processors;and one or more memories storing one or more programs configured to beexecuted by the one or more hardware processors, the one or moreprograms including instructions for: (1) acquiring a file including (a)material information for generating a virtual viewpoint image based on aplurality of captured images obtained from a plurality of imagecapturing apparatuses, the acquired material information includinginformation for specifying positions and orientations of the pluralityof image capturing apparatuses, wherein the positions of the pluralityof image capturing apparatuses differ from each other and theorientations of the plurality of image capturing apparatuses differ fromeach other and (b) additional information for specifying displayinformation that is included in the virtual viewpoint image generatedbased on the material information; and (2) generating a virtualviewpoint image including the display information based on the acquiredfile.
 16. The image generation apparatus according to claim 15, whereinthe additional information includes dependence information indicatingwhether or not display of the predetermined information on a virtualviewpoint image depends on a virtual viewpoint, and wherein (a) in acase where the dependence information indicates that the display of thepredetermined information depends on a virtual viewpoint, there isgenerated a virtual viewpoint image on which the predeterminedinformation is displayed on a condition that a virtual viewpoint is setto a predetermined line-of-sight position and a predeterminedline-of-sight vector; and (b) in a case where the dependence informationindicates that the display of the predetermined information does notdepend on a virtual viewpoint, there is generated a virtual viewpointimage on which the predetermined information is displayed irrespectiveof a line-of-sight position and a line-of-sight vector of a virtualviewpoint.
 17. The image generation apparatus according to claim 15,wherein the one or more programs further include instructions forsetting a line-of-sight position and a line-of-sight vector of a virtualviewpoint, and wherein the virtual viewpoint image is generated based onthe line-of-sight position and the line-of-sight vector of the virtualviewpoint.
 18. A file generation method comprising: a first acquisitionstep of acquiring material information for generating a virtualviewpoint image based on a plurality of captured images obtained from aplurality of image capturing apparatuses, the acquired materialinformation including information for specifying positions andorientations of the plurality of image capturing apparatuses, whereinthe positions of the plurality of image capturing apparatuses differfrom each other and the orientations of the plurality of image capturingapparatuses differ from each other; a second acquisition step ofacquiring additional information for specifying display information thatis included in the virtual viewpoint image generated based on thematerial information; and a file generation step of generating a filefor generating the virtual viewpoint image including the displayinformation, the file including (a) the material information acquired bythe first acquisition step and (b) the additional information acquiredby the second acquisition step.
 19. A non-transitory computer-readablestorage medium storing a program for causing a computer to execute amethod, the method comprising: a first acquisition step of acquiringmaterial information for generating a virtual viewpoint image based on aplurality of captured images obtained from a plurality of imagecapturing apparatuses, the acquired material information includinginformation for specifying positions and orientations of the pluralityof image capturing apparatuses, wherein the positions of the pluralityof image capturing apparatuses differ from each other and theorientations of the plurality of image capturing apparatuses differ fromeach other; a second acquisition step of acquiring additionalinformation for specifying display information that is included in thevirtual viewpoint image generated based on the material information; anda file generation step of generating a file for generating the virtualviewpoint image including the display information, the file including(a) the material information acquired by the first acquisition step and(b) the additional information acquired by the second acquisition step.